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POLYPHONIC DESIGN, CONDUCT, EXPERIENCE, AND 
EVALUATION IN CSCL CHATS 


Stefan TRAUSAN-MATU! 


Abstract. The paper presents an approach directed towards enforcing a polyphonic 
structure to a CSCL chat, with the aim of increasing inter-animation and collaboration. 
Another leading idea of the paper is that we should start from the analysis of the 
polyphonic experience of the learners, which occurs when they are situated in the 
presence of multiple collaborating voices in small groups. A careful designed assignment, 
in which learners are assigned roles (or ‘instruments’ in the musical perspective) can 
induce a polyphonic structure along both longitudinal and transversal directions, as in 
musical counterpoint. Another major consequence is that such a design facilitates the 
analysis of the collaboration and personal contributions of students. 


Keywords: Computer-Supported Collaborative Learning (CSCL), Polyphonic analysis, Discourse 
analysis, Natural Language Processing, Conversation Analysis 


1. Introduction 


Instant messenger (chat) has been used in recent years for Computer-Supported 
Collaborative Learning (CSCL) for various types of small group assignments 
[1,2,3,4]. Meanwhile, among the main theoretical foundations of CSCL were 
proposed the social-cultural theory and the view of language as a tool idea of 
Vygotsky [5] and Bakhtin’s dialogism [5,6,7,8,1,4]. 


Some developed CSCL systems offer interaction analysis tools of wikis and 
especially conversations, either spoken or through chat or forums: TATIANA [9], 
CORDTRA [10], VMT-Basilica [11], KSV [12], COALA [13], DIGALO [14], 
ColIAT [15], the Scaffold-Argument visualization [16], and Dong’s system [17], 
Some of these systems use several kinds of argumentation graphs, some of them 
in the idea of Toulmin [18], or more elaborated structures like the contingency 
graphs [19]. 


The approach presented in this paper is based on a totally different theoretical 
basis. It aims at identifying in chats a polyphonic structure and inter-animation 
patterns [4]. This theoretical framework was used as a starting pomt for the 
implementation of a series of systems for chat analysis: Polyphony [20], 
PolyCAFe [21], ReaderBench [22] and several others. The PolyCAFe system 
(Polyphonic Conversation Analysis and Feedback generation) was designed for 
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supporting both learners and tutors with abstraction and feedback-giving services 
for evaluating the assignments based on chat or forum conversations [21]. 
PolyCAFe is implemented using natural language processing tools for discourse 
and content analysis. Three classes of information are investigated: if the content 
of the utterances is semantically close to the topics supposed to be discussed, if 
there is a real collaboration (for example, the acceptance of others’ oppinions and 
inter-animation patterns) and if discourse and debate threads may be identified. 


The paper presents an approach directed towards enforcing a polyphonic structure 
to a CSCL chat, with the aim of increasing inter-animation and collaboration. 
Another leading idea of the paper is that, in CSCL chat analysis we should start 
from identifying polyphonic experiences of learners, similar to players in 
polyphonic music. A careful designed assignment, in which students are assigned 
roles (or ‘instruments’ in the musical perspective) can imduce a_ polyphonic 
structure along both longitudinal and transversal directions, as im musical 
counterpoint [4]. Another consequence is that such a design facilitates the analysis 
of the collaboration and personal contributions of students. 


The polyphonic assignment presented in the paper was used in the last five years 
at the “Politehnica” University of Bucharest in a course on Human-Computer 
Interaction (HCI) for the Computer Science and Engineermg students in their last 
year of undergraduate studies. The assignment was well accepted by the students, 
which participated with interest. It was also used for the validation of the 
PolyCAFe system in the LTfLL project (http//itfll-project.org/). The paper 
continues with an introduction to the polyphonic model, with a discussion about 
the experiencing of CSCL chats and the conducting of CSCL in a polyphonic 
manner. The following section presents the ideas of designing polyphonic chats. 
The facilities for evaluation of the polyphonic structure are presented in the end. 


2. The Polyphonic Model of CSCL 


Musical polyphony is a remarkable case of collaboration in which, starting from a 
theme, several “voices” concomitantly play it or its variations achieving a 
coherent joint piece. Each “voice” may be either a distinct instrument or group of 
instruments (a participant) or a sequence of notes played, together with other 
voices on a same instrument. For example, there are polyphonic musical pieces 
for 3-4 voices played by a single interpret at piano. The same piece may be also 
played by an orchestra where each voice is played by a group of mstruments. 
Therefore, there is not a one-to-one association between a voice and a participant. 
Musical polyphony may be achieved by composition in classical music on paper 
(e.g. the fugues of Johann Sebastian Bach) or by improvisation, with a lower 
degree of complexity in other types of music, as jazz and latin music. 
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The idea of considering a polyphonic model in natural language was introduced 
by Mikhail Bakhtin [7], which developed also the theory of dialogism, which sees 
everything as a dialog. Dialogism was proposed as a paradigm in CSCL [8] and 
together with the associated concept of inter-animation was used for analyzing 
CSCL chats [4,23]. 


The polyphonic model of CSCL group interaction considers, similarly with the 
musical case, that the students are experiencing the presence of a number of 
concurrent voices, each of them having independence but contributing to a joint 
discourse. The theme is the discussion subject or the problem to be solved 
assigned by the teacher or tutor. Each of the participants emits a coherent 
sequence of utterances and, interacting in a group, inherently has to solve the 
dissonances appearing between them. Therefore, participants face centrifugal and 
centripetal; convergent and divergent forces, along two directions: longitudinal 
and transversal, following the musical counterpoint rules [4]. 


However, these forces have an important effect, they allow the participants to 
perceive dissonances which put under question their utterances, they generate an 
inter-animation phenomenon. The polyphonic analysis tries to identify inter- 
animation patterns along the two dimensions and corresponding to the two types 
of forces. The longitudinal dimension may be found in references (links), which 
may be explicit or implicit and in repetitions of words which form threads which 
may be seen as parallel voices to which implicit voices (e.g. of the teacher, of the 
authors of the readings) are added. The transversal dimension may be generated 
by dissonances or differential positions among voices. 


2.1 Experiencing and Conducting the CSCL Polyphony 


The Polyphonic music is a complex process of developing a theme in a 
community of voices, which is validated by the reception of the listeners. It may 
be seen also as a model of creative thinking. For example, those who enjoy the 
fugues of Johann Sebastian Bach understand that sometimes his music is like an 
explosion of ideas. 


If we consider the polyphonic model of CSCL, attention should be paid to avoid 
that a cacophony result appears, where voices interfere in an disarmonic way. In 
fact, in classroom based education, or in group learning, professors usually 
conduct the discussions along a theme, on a longitudinal dimension. Meanwhile, 
many times they stimulate learners to consider transversal dimensions, for 
discovering differences that drives to the concepts that should be learned. 


The dialog in the classroom or in learning groups is many times directed towards 
constructing a coherent discourse driving to the concepts to be learned or to 
experience the needed skills (for example the reasoning in mathematics problem 
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solving). The role of dialog was emphasized by Sfard, which defined learning not 
as a transfer of knowledge, but rather as “becoming a participant in a certain 
discourse” [24]. Therefore, it is extremely important to placing the learners in the 
position of experiencing and contributing to dialogue. 


Starting from the idea of Mikhail Bakhtin that everything is a dialog, individual, 
internal thinking of learners is also a dialog and should also be considered for the 
analysis of the conversation. Moreover, means of inducing it are beneficial. In 
fact, when teachers encourage students to put themselves questions and _ find 
answers, this is, in fact, an implicit urge to internal dialog. 


One important feature of listening or contributing to polyphonic music or, in our 
CSCL perspective, to contribute to group knowledge construction, is the 
perception, the following and the participation to the threads (voices) of 
discussion in dialogs. 


In a face-to-face conversation, due to the impossibility that two persons speak in 
the same time, there is very difficult to handle more than a single thread of 
discussions. This is not the case when using chat environments like the one of the 
Virtual Math Teams (VMT) project [2], which offers explicit referencing 
facilities. By this facility, learners in groups may indicate to which previous 
utterance(s) they refer to, allowing the existence of several discussion threads (or 
voices), in parallel. 


The co-occurrence of several threads generates inter-animation, a general feature 
of CSCL in chats [4]. Participants to CSCL chats face the need to handle many 
threads in parallel. The threads are formed by the explicit referencing facilities 
(when the VMT chat environment is used), by the temporal sequence of 
utterances, the implicit links of co-references, the adjacency pairs, or, at another 
abstraction level, transacts [25]. In addition, one important source of thread 
generation is the repetition of words, which has a very important role in inducing 
the sense of participation [26]. 


In Figure 1 several threads of the kinds discussed above are emphasized (curly 
arrows represent explicit references). To these threads/voices we should add other 
implicit ones, generated by the subject of discussion, teacher’s voice (even in 
absence) and by the implicit assumptions of each participant. 


For example, Tim, at utterance 26 should take into account the threads generated 
by the repetition of the words “topic”, “reply’, “double-clicking”, “presentation”, 
those of explicit referencing, and the implicit thread of the subject of the CSCL 
assignment (to discuss ways of developing collaborative environments), which 
may be considered as the voice of the teacher. As a consequence, at utterances 34- 


36 he contributes to several threads of the dialog. 
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Ref Time User Text 


10.26.37 adrian First of all, fhe| 
_ ike menay th 


10.28.15 adrian i'd 
10.28.38 adrian ormaybe multiple 
‘but itis reall 


P30 10.29.20 adrian 
31 10.29.47 adrian 


10.31.00 adrian 
10.31.44 adrian 


Fig. 1. An image of a subset of the threads in a conversation. 
3. Designing and Detecting Polyphonic Dialogism in Chatting Virtual Groups 


3.1 A Case Study 


CSCL chats has been used for the last ten years at the “Politehnica” University of 
Bucharest in a course on Human-Computer Interaction (HCI) for the Computer 
Science and Engineering students in their last year of undergraduate studies. More 
than 90 students enroll each year for this course, the maximum of 324 students 
being reached in the 2008-2009 year (when was a special situation because two 
generations completed the studies). 


During the course, students had lectures where they were taught the theory of 
interface design and evaluation. They had also laboratory hours and homework 
aiming to apply their theoretical knowledge by individually developing several 
applications. 


An important emphasis is given in the course to collaborative applications. In 
order to enhance the quality of their learning, we designed an assignment using 
CSCL chats where they discussed about collaborative technologies (forums, chat, 
blog and wiki). 
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3.2 Polyphonic Design of CSCL chats 


For the increase of the mterest and participation of the students, we started in the 
design of the scenario of the CSCL chat assignment from a_ polyphonic 
perspective, in which inter-animation is induced. Students were divided in groups 
of four and each of them was assigned a role: They were told to consider that they 
were the directors of a company selling one of the collaborative technologies 
presented at the course (chat, forum, blog and wiki). Before the chat they were 
supposed to individually study collaborative technologies and after that to have a 
1-2 hours chat. In the first part of the chat conversation, each student had to 
champion the technology he represented by presenting its features and advantages 
and criticize the others by invoking their flaws and drawbacks. In the second part 
of the chat, they had to discuss on how they could integrate all these technologies 
in a single online collaboration platform. Therefore, in a single conversation the 
students first engaged into a debate where a differential position should be taken. 
The results of the discussion in the first part were then used for collaboratively 
building a solution to the given problem. The VMT environment [2] was used due 
to its features of facility of inserting explicit references to previous utterances. The 
chat logs were analyzed by the tutors for evaluating and grading the activity of the 
students in the assignment. 


3.3 The Data 


For the purposes of validation of our approach we have selected two chat logs, 
one that we considered a good collaboration (Team36) and one (Team34) which 
was an example of a bad collaboration, with a lower degree of mter-animation. 
The chat logs of the two teams of students were evaluated and graded by four 
tutors (T1, T2, T3 and T4) and by an automatic grading system [4]. They gave 
marks from | to 10 (10 is the best) for each student (see Tables 1 and 2) and they 
also wrote some feedback. Tutors were also asked to report what were the 
problems they faced. 


Table 1. Team34 


pT 2 3 | TH | Average | Auteval | 
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Table 2. Team36 
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3.4 The Polyphonic Analysis 


The polyphonic analysis was performed manually with the visualization facilities 
of PolyCAFe. The longitudinal and transversal dimensions were analyzed, with a 
focus on threading. The polyphonic design has been facilitating the analysis 
because we knew what ‘instrument’ played each learner. 


The Longitudinal Dimension - Referencing 


One first image of the degree of inter-animation in a chat is the usage of explicit 
and implicit references along the longitudinal dimension. The explicit references 
are those indicated by the learner when using the VMT system [2]. The implicit 
references are detected by the instruments we have developed in PolyCAFe. 


One idea of analyzing the degree of participation of a student is to start from the 
assumption that if she uses many explicit references and the system detects also 
many implicit ones, she displays a collaborative attitude. For example, it is 
obvious from seeing the Figures 2 and 3 that the participants in Team34 are linked 
by much lesser references than those in Team36, fact which is consistent with the 
evaluations of the tutors. 
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Fig. 2 References of Team34 (dots are utterances; blue lines are explicit references and yellow 
lines are implicit references; numbers are utterances’ numbers; time flows from left to right). 
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Fig. 3. References of Team36. Blue lines are explicit references. 
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Fig. 4. References of Team36. Blue lines are implicit references. 


An analysis of the distribution of implicit and explicit links (the difference 
between Figure 2 and 3) shows that they are in the majority of cases distinct. For 
example, in the Team36 example, only one link was common. This shows that the 
learners are aware of the implicit links and they also feel the need of assuring the 
threading. Therefore, they use explicit links only when they feel that the threading 
could not be obvious. However, a problem with this kinds of diagrams and 
analysis is that, even if they can give an image of the degree of collaboration, 
more precise analysis should be performed in order to detect zones of good 
collaboration or to identify real, important, contributions. For example, one may 
use a lot of explicit references, or may write utterances that generate implicit 
utterances, but these might be insignificant (e.g. only formal approvals or 
unjustified denials of the others’ utterances). A solution to this problem is_ the 
analysis of the other features of a good polyphony: threading and transversal 
utterances. 
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Fig. 5. The threads of the assigned topics in Team36 (up) and Team34 (down). 
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Threading 


The majority of approaches for chat analysis mentioned in the mtroduction are 
based on the analysis of pairs of utterances. One of the main ideas of the 
polyphonic model is the emphasis on threads and their interactions. For this aim 
we have developed a facility of visualiznmg threads of repeated words in 
PolyCAFe [21]. For example, in Figure 5 you can see comparatively the threads 
of the assigned concepts to be discussed (chat, blog, forum and wiki). The fact 
that in Team34 there is less variation signifies that the collaboration was not so 
good as compared to Team36. 


The visualization of the threading of additional concepts proved also very useful 
to see the polyphonic structure and the contributions of the participants. These 
additional concepts were detected from the analysis of the most used concepts in 
the chats. For example, in Figure 6 is presented the threading of the most used 
concepts in Team36 (besides the assigned topics — chat, forum, blog and wiki). 
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Options Conversation thread Special threads 


Ignores the implicit and explicit links and shows other patterns and threads that you specify 


Input concepts to search for (separated by commas, ): 
(e.g. concept1, concept2): 
Use concepts from assignment 


time Word colors: 
people, Word 1: time 
information, Word 2: people 


Word 3: information 


Fig. 6. The threading of some additional concepts in Team36. 


The Transversal Dimension - The Differential Positioning 


Even if explicit and implicit references and the threads of repeated words may 
give an image of the contribution of each learner, they are not covering all the 
aspects of the collaboration, of the inter-animation. Therefore, some other analysis 
methods are needed. Moreover, an analysis of the performance of each learner is 
needed. 
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The fact that the chats were designed from a polyphonic perspective, based on 
assigning roles is not only inducing polyphonic order, it also facilitates the 
implementation of analysis and feedback giving because we can more easily 
identify the utterances where a participant takes a differential position discussing 
about others’ roles. 


From the polyphonic, counterpoint perspective, we should consider both the 
longitudinal dimension, when a student discuss about the personal assigned topic 
and the transversal topic, when she poses herself in a differential position, 
discussing about others’ topics. For example, in Figure 7, utterances 74, 75, 76 are 
differential, while 75, 76, 84, 85, 86, 88 and 89 are longitudinal. 75 and 76 are 
utterance combining the two features. Utterance 74 is an example of an utterance 
which refers to utterances which contain other topic. 


r |Student Utterance 
if you a willing to spend a lot of time, that may be true for a small wiki , but when you a trying to develop an encyclopedia ....you 
have absolutely no chance 
Messages posted to a forum are publicly available for some time, which is uncommon in chat rooms, with only few exceptions. 


why are blogs search-engine friendly? I think wikis are more search-engine friendly! :P 


chats work as long as you have an internet connection... 


because you can use keywords 

never say never - what about Wikipedia ?! 

can you detail on that feature a bit? 

in terms of 'coding' your application, I bet yours is the hardest 

and only that could take up a lot of time 

it may be, but you're only going to do it once....and then use it like that for a very long time 


the blogs can be grouped by interest and all the articles can be full of keywors that are search engine friendly 
do not forghet that a blog is the best way to promote a site 
same with forums, and reading the posts looks easier 


ok, thanks for the info 


sorry , but the est way is a message to all your friends..using a CHAT 
but if you want to study, you have a lot of information on the forums, moreover, if you have questions people can take their time to 
answer you 


Fig. 7. Evidentiation of the transversal positionings. 


In Figure 7, each student has an associated topic (iv), in our case one of chat (C), 
forum (F), wiki (W) or blog (B). She says something that may contain one or 
more of the associated topics (cv) of her or colleagues. She may explicitly refer to 
another utterance that may also contain some topics (rv). 


There are several situations that may appear, when there are topics found in the 
utterance (cv) or in the referred utterance (rv). The student: 


1. iv=cy - speaks about her personal topic 

2. iv=rv - explicitly refers an utterance containing a topic that is her topic 

3. cv=rv - speaks about the same topic with the one discussed in the utterance 
she explicitly referred 
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4. iv#cv - speaks about other topic than her own 

iv#rv- refers about an utterance containing a topic that is not her own 

6. cv#rv - speaks about a different topic that the one discussed in the 
utterance she explicitly referred 


om 


The analysis of the number of occurrences of some of the above situations and of 
some of their ratios may give an image of the contribution of the participants to 
collaboration and inter-animation. For example, a good contribution should have a 
balanced ratio between the utterances oriented towards longitudinal, selfish, on 
own topic and transversal, directed towards others’ topics. In our opinion, a value 
of nfjyvzcy/Mjy=cy Close to 1 is characterizing such a situation. 


The results of the analysis of Team34 and Team36 using the above terms is 
displayed in Tables 3 and 4. The relative values of nfivcy/Mtiyacy are rather 
corresponding to the relative appreciations of the tutors in the classification of the 
learners. 


Table 3. Team 34 


; oe > fdat. caine Average 
Nut NYjy=cy NY ivzev DN ovzrv NY jy=cy/DTiy NY ivzev NTjy NY ivtev' Niv=cyv tutors’ mark 


Table 4. Team 36 


oe 
Nfut NTiv=cv NTiv4ev NlevAry Ni jy=cy/NTjy Nfpy¢rv/ NT jy 4 a5 | 
Ee a a z 


ee 
Fe Eee ea (ee 


peer fpf pf foe oe er 


In tables 3 and 4: 


Nyt is the number of utterances emitted by the student 

NFiy=cy IS a lower bound of the number of utterances in which a student speaks 
about her personal topic 

NFiydey 1S a lower bound of the number of utterances in which a student speaks 
about a topic other than her own 

NFiy#ry 1S a lower bound of the number of utterances in which a learner referred 
a topic which is not her own 
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4. Conclusions and Further Developments 


The polyphonic structuring of a CSCL chat can be conducted by a careful design, 
in which each learner is assigned a role that is conflictng with the other 
participants in the first part of the conversation, but in the end it should reach a 
consensus. The fact that the ‘interests’ of each learner are assigned by the teacher 
simplifies also the evaluation of the collaboration and of the participation of the 
learners. However, these tasks are challenging and the existence of a series of 
support tools and an evaluation methodology is beneficial. 


The first validation showed that the results of the polyphonic analysis are close to 
those of tutors which used only reading for the evaluation. In the future, the 
methodology will be supported also by natural language processing tools that could 
analyze in more detail the differential positioning of the utterances and the 
dissonances that appear between voices. Another development is the further analysis 
of the polyphonic experience of the learners, including also the investigation of the 
internal dialogues induced by the utterances from the external one. 
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